Investigation and Reduction of Discretization Variance in Decision Tree Induction

This paper focuses on the variance introduced by the dis-cretization techniques used to handle continuous attributes in decision tree induction. Diierent discretization procedures are rst studied empirically , then means to reduce the discretization variance are proposed. The experiment shows that discretization variance is large and that it is possible to reduce it signiicantly without notable computational costs. The resulting variance reduction mainly improves interpretability and stability of decision trees, and marginally their accuracy. Decision trees ((1], 2]) can be viewed as models of conditional class probability distributions. Top down tree induction recursively splits the input space into non overlapping subsets, estimating class probabilities by frequency counts based on learning samples belonging to each subset. Tree variance is the variability of its structure and parameters resulting from the randomness of the learning set; it translates into prediction variance yielding classiication errors. In regression models, prediction variance can be easily separated from bias, using the well-known bias/variance decomposition of the expected square error. Unfortunately, there is no such decomposition for the expected error rates of classiication rules (e.g. see 3, 4]). Hence, we will look at decision trees as multidimensional regression models for the conditional class probability distributions and evaluate their variance by the regression variance resulting from the estimation of these probabilities. Denoting by ^ P N (C i jx) the conditional class probability estimates given by a tree built from a random learning set of size N at a point x of the input space, we can write this variance (for one class C i) : (1) where the innermost expectations are taken over the set of all learning sets of size N and the outermost expectation is taken over the whole input space. Friedman 4] has studied the impact of this variance on classiication error rates, concluding to the greater importance of this term as compared to bias. Sources of Tree Variance. A rst (important) variance source is related to the need for discretizing continuous attributes by choosing thresholds. In